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ABSTRACT 

Coded exper ience and effective semiotic systems, 
organizations of patterns of signs that comprise a system of meaning, 
have proved essential for the propagation of knowledge. In education, 
communication via semiotic systems forms an important role in guiding 
children toward the effective use of information and techniques, and 
it is important that children be able to decode a given 
representation competently. As semiotic devices, computers have 
additional power in that not only can they externally present 
representations, but they can also internally manipulate them. Three 
areas have been identified where their use may enhance learning: the 
representation of complex structural domains, the navigation of a 
lexical space depicting the relationships between words, and the 
display of a hypertext structure in an authoring tool. Three systems 
were designed to investigate the use of interactive schematic 
representations for these learning activities and include browsing 
the representation of a structured domain, navigating the lexical 
space, and accessing the structure of a story in an interactive 
fiction authoring tool. The aspect of these systems that 
distinguishes them from other computer educational systems which use 
schematic representations is the fact that what is represented is 
declarative subject matter, rather than the basis for a simulation 
tool or a procedural exercise. (Contains 14 references.) (AEF) 
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Helen Purchase 
Computer Science Department, University of Queensland, Australia 

Abstract: This paper reviews semiological theories in relation to the use of com- 
puters for enhancing learning activities, concentrating particularly on the use of 
graphical (or schematic) texts in declarative domains. Relevant semiological and 
educational issues are discussed, and three areas where the use of schematic rep- 
resentations in computer learning environments may be investigated are proposed: 
representing a complex structural domain, depicting the lexical relationships in a the- 
saurus, and assisting the authoring of an interactive fiction system. Three existing 
systems which address the associated learning activities are also described. 



Introduction 



Literacy is a primary aim of education: subject matter is usually taught ^out of context' with 
the use of various forms of text, and it is considered essential that children learn the skills to 
decode represented experience competently so that the information and techniques which form 
part of a culture may be propagated. Emphasis is therefore placed on reading, numeracy and 
the interpretation of diagrams and film. The computer is a significant addition to this list: its 
power as a representational device is greater than other media because of its interactive and 
processing nature. This paper considers the use of computers in education from the point of 
view of representational semiotic devices. First, relevant semiotic terminology 'is defined and ap- 
propriate educational issues addressed. Three projects which entail the design and development 
of computer educational systems which use an interactive representational method are then de- 
scribed. The aim of each of these systems is to investigate the suitability of using schematic 
representational methods for the depiction of declarative subject matter in an interactive device. 
They are each related to a particular learning activity: envisaging the structure of a complex 
domain, discovering the relationships between words in a lexicon, and authoring a hypertext 
system. 



Semiology: terminology and classifications 
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Definitions 

Semiology was defined by Saussure in 1915 "A science that studies the life of signs within 
society. . .[to] show what constitutes signs, what laws govern them. . (Saussure, 1959). While 
his emphasis was primarily linguistic, Peirce's analysis of signs (which he termed semfofics) was 
ino.-e general and concentratcMl on the characteristics of all signs (Peirce, 1977), Terminology 
differs between Saussure's and Peircc's definitions of signs and symbols, and for the purposes 
of this document, Pcirce's flcfinitioiis and categorisations will be used; that is, a sign is an 
intimate relation between an object (term) and an interpretant (concept). While the nature of 
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the bond between tlic two cojuponeuts of a sign is irrelevant, a symbol is a particular category of 
sign, where the relationship between its object and interpretant is arbitrary. A semiotic system 
(code) is an organisation of patterns of particular signs (usually rule- and convention-based) 
that comprises a system of meaning, and a symbolic system is a type of semiotic system. A 
semiotic system thus consists of a syntax which defines the manner in which terms may be 
organised, and a semantics which indicates how meaning can be attributed to a syntactically 
correct pattern of terms. A statement is defined as a syntactically correct and meaningful 
combination of terms in a semiotic system, a message as a semantically coherent sequence of 
statements, and text (representation) as the physical realisation of a statement or message. A 
representational system (or medium) is an abstract term that refers to the physical realisation 
of the rules and conventions that comprise a semiotic system, and a device is a physical object 
used for communication via a semiotic system. Communication takes place when text is created 
according to a particular code, and transmitted via a device. The receiver of the text decodes it 
to extract meaning, choosing to impose a particular code on the text to enable interpretation. 

For example, music is a semiotic system comprising signs (notes), where each note consists of 
•a term {eg E flat) and a concept (the sensation produced by the corresponding sound wave). 
A figure from Beethoven's ninth symphony is a statement, the whole symphony is a message, 
the score is a text, an orchestra is a device for communicating it, and musical notation is the 
abstraxt term denoting the corresponding representational system, 

A distinction may be made between non-interactive communication devices, which do not al- 
low the receiver of a text to interact with then {eg televisions), and interactive communication 
devices, whose flexibihty allows text to be altered and manipulated by the receiver. The most 
obvious example of an interactive communication device is a computer: its flexibility allows for 
dynamic transmission and manipulation of signs and texts. 

Classifiications 

Bruner identifies three difi^erent types of semiotic systems: enactfve, iconic and symbolic, with 
the corresponding signs action, image and symbol (Bruner, 1966). The enactive system is based 
on physical movement and the learning of responses {eg riding a bicycle), the iconic system 
depends on imagery and perception (e^ looking at a picture), and the symbolic system uses 
symbols which do not have a perceptual relationship with the objects they signify (e^ reading a 
sentence in natural language). For the purposes of this document, symbolic systems are further 
classified into schematic and linguistic categories, the main diff'erence between them being the 
way in which the symbols are physically organised. Schematic systems use spatial indicators 
to show the structure of the subject matter (eg circuit diagrams, flowcharts), while linguistic 
systems place the symbols in a purely sequential manner (e^ formal logic, morse code). In both 
systems, the way in which the symbols in a text are organised is important: linguistic systems 
have a linear syntax which determines how the meaning of the separate symbols unite to produce 
the semantics for the whole statement. Schematic systems tend to be graphical in nature, the 
overall structure of the subject represented and the relationships between the individual symbols 
being interpreted according to a conventional code (e^ straight lines in circuit diagrams represent 
physical connections which are not necessarily straight). The physical realisation of a statement 
or message in a schematic system is here termed a diagram. Schematic representations are 
exceptionally useful. Some concepts cannot be captured by icons or individual symbols alone 
and the hoHstic nature of diagrams ensures structure and relationships are made explicit, often 
making a simple diagram more effective than the equivalent linguistic form. 
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Semiotic skills and the educational use of computers 



The use of semiotic systems is mankind's most distinctive characteristic; coded experience and 
effective semiotic systems have proved essential for the propagation of knowledge. In education, 
communication via semiotic systems forms an important role in guiding children towards the 
effective use of information and techniques, and it is therefore important that children be able to 
decode a given representation competently; they should know the appropriate code well enough 
to both understand the text as well as manipulate and use it. For this reason, the development 
of literacy in various semiotic codes is the primary concern of much formal schooling. 

Subject matter is usually taught to children 'out of context' by the use of a (usually symbolic) 
medium. Olson and Bruner make a clear distinction between the knowledge represented in a 
text and the skills required to decode the text. They consider that, regardless of whether the 
medium is enactive, iconic or symbolic, a common invariant 'deep structure' of knowledge can 
be created. However, the skills used for extracting and utilising this knowledge in each medium 
differ greatly, and they recommend that this should be considered when a representational 
system for instruction is chosen: 

The choice of a means of instruction, then, must not depend solely upon the effec- 
tiveness of the means for conveying and developing knowledge; it must depend as 
well upon its effects on the mental skills that are developed in the course of acquiring 
that knowledge (Olson and Bruner, 1987). 

Tie choice of medium for encoding subject matter for educational purposes is therefore an 
important issue on two counts. Firstly, the ability of the intended users to understand the code 
in order to perform the required task should be considered. This is related to the cognitive 
development of the children and their previous experience with similar systems. Secondly, the 
suitability of the medium for the subject matter embodied and the intended learning activity 
ought to be addressed: inappropriate use of a medium may result in the creation and use of 
unsuitable and confusing mental models of the domain. 

Both of these issues are made more complex when the computer is chosen as the medium of 
instruction. As semiotic devices, computers have additional power in that not only can they ex- 
ternaUy present representations, they can also internaUy manipulate them. Educational studies 
regarding the semiotic nature of the computer tend to be concerned with the cognitive benefits 
accrued from programming (eg understanding random numbers, recursion, variables etc) (Mc- 
Dougall, 1990). They have concentrated on the symbolism afforded by programming languages 
(Papert, 1980; Pea, 1987), and have not addressed the computer's potential as a semiotic inter- 
active device for the explicit representation and manipulation of information, where no enactive 
(ie simulation or program) counterpart exists. The emphasis in the practical educational use of 
computers is usuaUy on the subject matter embodied within the software, or on the computer 
skills re-inforced by computer use, rather than on the nature of the medium used. 

Three interactive schematic systems 

In focusing on the semiotic nature of the computer as a representational device, educational 
researchers and practitioners ought to consider whether its interactive nature contributes in 
any way to the Icarninn activities that the user is engaged in (or indeed, whether it inhibits 
them). The research reported here considers interactive declarative schematic representations 
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in particular, and three learning activities have been identified as potential areas where" their 
use may enhance learning: the representation of complex structural domains, the navigation 
of a iexical space' depicting the relationships between words, and the display of a hypertext 
structure in an authoring tool. 

Three systems have been designed to investigate the use of interactive schematic representations 
for these learning activities: they are described briefly below, with particular emphasis given to 
the nature and use of each the schematic representations used. 

Schematic hierarchical knowledge base structures 

Kreek is an educational learning environment which embodies an interactive schematic repre- 
sentational method for the portrayal of structured declarative domains {eg the solar system, an 
animal taxonomy), using a network syntax where nodes represent objects and links represent 
relationships. Relationships between the main objects in a text are made explicit by forming 
links between them, while attribute (or 'property') information associated with the objects are 
encapsulated within frames at the nodes (Minsky, 1975). In addition, each node may have 
associated with it a layer, which is itself a syntactically valid diagram and covers the entire com- 
I)uter display. Thus Kreek texts may consist of any number of layers organised in a hierarchical 
structure, allowing the knowledge-base to be structured so that subordinate layers provide more 
specific information about an object.^ The dynamic environment offered by the computer is 
therefore used in order to incorporate structural conventions: not all of the information is made 
available at once, and the parts of the text that are not initially visible (either in frames or in 
layers) may be selected by the user for further perusal. Kreek may be used for both browsing 
and creating these hierarchical schematic texts: the intention is that the inherent structure of a 
knowledge base may be better envisaged by making it exphcit in an interactive schematic text. 

Quahtative preliminary evaluation of this system with nine-year old children revealed that their 
ability to recognise and use appropriately the structural conventions embodied within a K REEK 
text varied with the mode they were in. They had little difficulty in understanding the fact 
that the positioning of the facts within the hierarchy provided additional information when they 
were browsing di Kreek text, but only the higher-ability children used these structural conven- 
tions appropriately in authoring mode (Purchase, 1992). Further experimentation needs to be 
done to supplement these observations with a more rigorous qualitative analysis, encompassing 
a larger sample size, and focusing more particularly on the structural envisagement aspect of 
text browsing and ceation, and its support of the learning activity. 

Depiction of ^lexical space' 

Glgcs is currently under development: it is intended to be a language tool which portrays the 
highly-linked information in a thesaurus graphically, providing the means for easy and efficient 
access of the entries for the words cross-referenced by a particular lexical entry, and displaying 
these relationships between the entries in a graphical manner. Hypertext type implementations 
of thesauruses have implicit links between the entries: the system proposed here will make these 
relationships graphically explicit, thus allowing the user to visuaHse the 'lexical space' in which 
the word is situated. Thus the Mexical area' defined by a word and its lexical associations will 
be depicted in a graphical manner (using a node-arc notation similar to QuiUlan's ^semantic 

'This idea is similar to Quillian's linking together of planes (Quillian, 1967), without the condition that hnks 
between planes be restricted to relationships between concepts and their corresponding instances. 
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memory' representation (Quillian, 1967)), in order to allow the relationships between the words 
and their cross-references be perceived visually. A lexical entry will be visually depicted as 
a 'focus' node representing the word, which is joined by graphical arcs (indicating lexical' or se- 
mantic relationships) to the relevant cross-referenced words. The display is made more complex 
by the further arc-node depiction of words associated with these cross-references. User selection 
of any word which is displayed on the screen in one lexical entry would change the graphical 
display so that the newly selected v;ord forms the focus of the display. The new focus node, its 
cross-referenced words, and further lexical or semantically associated words would now be dis- 
played. By appropriate selection of words, the user can browse through the thesaurus, following 
relationships through the lexical space. A system like this may have many uses in education, in 
learning about the structure of the lexicon (either for children or for people learning English as 
a second language), or for discovering the semantic relationships between concepts. In addition, 
the graphical aspect of Gloss provides any user with a new way of visualising words, semantic 
and lexical relationships and lexicon structure. 

Schematic structures for interactive fiction 

Storytime is a creative writing authoring tool, which allows non-sequential stories to be both 
read and written. It maintsiins a hypertext type implementation of events in a story, with the 
links between the events relating to choices that the reader makes when decision points are 
reached in the narrative. The non-sequential nature of the stories is what is of most interest 
from both a reading and an authoring perspective, and this underlying structure of highly-linked 
nodes allows many stories with different conclusions to be explored and created. Storytime 
is intended to be used by prim.ary school children in both reading and authoring mode, and it 
provides a simple interface for both activities. 

Readers of STORYTIME stories suffer from the problems that users of hypertext systems are 
prone to: not knowing where in the linked structure they are currently located, and having 
no prior knowledge of the scope of the story (in terms of both size and possible events in the 
narrative).^ When creating stories, authors may experience the same problems, but these may 
be exemplified by errors or omissions that they may have made {eg forgetting to ensure that 
all pages are accessible, associating the wrong page with a link, or losing sight of the overall 
structure of the story, and the various narratives that it may entail.) 

Storytime has been partially extended in an attempt to address these problems by including 
the facility for the overall structure of the story to be viewed as a schematic display of a directed 
graph. Thus the internal hypertext representation of nodes and links is made visible to assist 
the reader in knowing something more about the structure and entent of the story. In addition, 
the author will be able to get an overview of the possible paths in the story, and ensure that 
all pages are accessible and the created structure correctly represents their intention. Further 
implementation is necessary to provide graphical authoring facilities. 

Like the thesaurus implementation discussed above, these extensions to STORYTIME require 
the use of a suitable automatic graphical display algorithm which will display the structure of 
the highly-linked internal representation of nodes and links. Graphical tools which implement 
algorithms which enable the perspicuous display of graphs {eg the 'spring' algorithm (Kamada, 
1989)) wiU be used. 

roador of a book may gut bouic icIca uf its scope by srcing how thick it is, or by ''flicking tliough" to get some 
preliminary notion of the nature ol the language and events that can be expected. There is no similar equivalent 
in StofIYTIME. 
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Conclusion 

The three systems described above are intended to be used to investigate the potential for the 
use of schematic representations in an interactive device for three particular learning activities: 
browsing the representation of a structured domain, navigating lexical space, and accessing the 
structure of a story in an interactive fiction authoring tool. The aspect of these systems that dis- 
tinguishes them from other computer educational systems which use schematic representations 
is the fact that what is represented is declarative subject matter, rather than the basis for a 
simulation tool (Cheng, 1993), or a procedural exercise (Levonen and Lesgold, 1993). Successful 
use of these systems (ie the enhancement of the respective learning activities) would demon- 
strate the suitability of this choice of medium, and would indicate that the interactive nature of 
the computer can indeed be exploited with schematic representations for the effective depiction 
of declarative subject matter. 
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